Multigrid Q-learning

Reinforcement learning scales poorly when reinforcements are delayed. The problem of propagating information from delayed reinforcements to the states and actions that have an e ect the reinforcement is similar to the problem of propagating information in a discretized boundary value problem. Multigrid methods have been shown to decrease the number of updates required to solve boundary value problems. Here we extend Q-Learning by casting it as a multigrid method and show a reduction in updates required to reach a given error level in the Q-function for a simple, 1-d Markov decision task.